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A GLOBALLY CONVERGENT AUGMENTED L AGRAN GIAN PATTERN SEARCH 
ALGORITHM FOR OPTIMIZATION WITH GENERAL CONSTRAINTS AND SIMPLE 

BOUNDS 


ROBERT MICHAEL LEWIS * AND VIRGINIA TORCZON t 

Abstract. Wc give a pattern search adaptation of an augmented Lagrangian method due to Conn, 
Gould, and Toint. The algorithm proceeds by successive bound constrained minimization of an augmented 
Lagrangian. In the pattern search adaptation wc solve this subproblem approximately using a bound con- 
strained pattern search method. The stopping criterion proposed by Conn, Gould, and Toint for the solution 
of this subproblem requires explicit knowledge of derivatives. Such information is presumed absent in pat- 
tern search methods; however, we show how we can replace this with a stopping criterion based on the 
pattern size in a way that preserves the convergence properties of the original algorithm. In this way we 
proceed by successive, inexact, bound constrained minimization without knowing exactly how inexact the 
minimization is. So far as we know, this is the first provably convergent direct search method for general 
nonlinear programming. 

Key words, augmented Lagrangian, constrained optimization, direct search, nonlinear programming, 
pattern search 

Subject classification. Applied and Numerical Mathematics 

1. Introduction. In this paper wc consider the extension of pattern search methods to nonlinearly 
constrained minimization. We will consider problems of the form 

minimize f(x) 

(1.1) subject to c(x) — 0 

£ < x < Uj 

where / : IR n — > 1R and c(x) = (ci(x), • • ■ , Cm{x)). We allow the possibility that some of the variables are 
unbounded either above or below by permitting £j,Uj = ±oo, j 6 {l,'--,n}. This formulation assumes 
that any general inequality constraints have been converted into equality constraints by the introduction of 
non- negative slack variables, leaving bounds as the only explicit inequality constraints. 

The pattern search method that we will discuss here is an adaptation of an augmented Lagrangian 
method due to Conn, Gould, and Toint [6]. The latter method is the basis for the subroutine AUGLG in 
the LANCELOT optimization package [7]. The method of Conn, Gould, and Toint involves successive bound 
constrained minimization of an augmented Lagrangian. Since pattern search methods have recently been 
extended to bound constrained minimization [19, 21], an adaptation of the augmented Lagrangian method 
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of Conn, Gould, and Toint to pattern search naturally suggests itself. Furthermore, the multiplier update 
of Algorithm 1 in [6] does not involve information about derivatives of the objective or constraints, so the 
augmented Lagrangian approach is consistent with the derivative- free nature of pattern search algorithms. 

Since there exist broad classes of pattern search methods for unconstrained [20, 34, 35, 36] and bound 
constrained minimization [19, 21], it seems to us natural to first extend pattern search methods to nonlinearly 
constrained minimization via algorithms that proceed by successive unconstrained or bound constrained 
minimization, such as the augmented Lagrangian method we discuss here. In the absence of information 
about derivatives of the objective and constraints, it is difficult to design pattern search algorithms for 
general nonlinearly constrained minimization that produce only feasible directions or feasible iterates. This 
is due to the fact that a pattern in a pattern search algorithm would need to include a sufficiently rich set 
of search directions to capture any feasible improvement in the objective. When nonlinear constraints are 
present, it is not clear how to design such a pattern without first-order information. 

We will show that despite the absence of an explicit estimation of any derivatives (a characteristic of 
pattern search methods), our pattern search augmented Lagrangian approach exhibits all of the first-order 
convergence properties of the original algorithm of Conn, Gould, and Toint. This at first is surprising, since 
the original algorithm allows its subproblems to be solved approximately, and the stopping criterion for the 
solution of the subproblcms is based on the magnitude of a measure of first-order stationarity for bound 
constrained minimization. This information is not explicitly available in a direct search method. However, as 
we discuss in §5.1, there is a correlation between the size of the pattern in bound constrained pattern search 
and the amount of local feasible descent. Using this correlation we are able to establish convergence even 
without explicit knowledge of derivatives. That is, we are able to proceed by successive, inexact minimization 
of the augmented Lagrangian via pattern search methods, even without knowing exactly how inexact the 
minimization is. 

This is the main contribution of the work presented here, and shows how one can use pattern search in 
a practical algorithm for nonlinear programming. Otherwise, the extension of pattern search to constrained 
minimization by means of the augmented Lagrangian approach of Conn, Gould, and Toint is straightforward, 
due to the strength and generality of the convergence analysis presented in [6], 

The question of treating general nonlinear constraints with direct search minimization algorithms has 
a long history, beginning with the original work on direct search methods. Rosenbrock, in [28], proposed 
treating constraints using his rotating directions method by redefining the objective near the boundary of 
the feasible region in a way that would tend to keep the iterates feasible, a form of penalization. Similar ideas 
for modifying the objective in the case of bound constraints are discussed by Spendley, Hext, and Himsworth 
[30] and Nelder and Mead [24] in connection with their simplex-based methods. In these approaches the 
objective is given a suitably large value (in the case of minimization) at all infeasible points. 

More systematic approaches to penalization have also appeared. The treatment of inequality constraints 
via exact, non- smooth penalization (though not by that name) appears as early as the work of Hooke and 
Jeeves [15]. More recently, Kearsley and Glowinski [13, 16] have applied pattern search methods to equality 
constrained problems arising in control via exact, non- smooth penalization. Weisman’s MINIMAL algorithm 
[14] applies the pattern search algorithm of Hooke and Jeeves to a non-smooth quadratic penalty function and 
incorporates an element of random search. Davies and Swann [8], in connection with applying the pattern 
search method of Hooke and Jeeves to constrained optimization, recommend the use of the reciprocal barrier 
method of Carroll [5, 11]. 

A direct search method for constrained minimization that has proven very popular in application is 


M. J. Box’s Complex method [3], which was originally developed to address difficulties encountered with 
Rosenbrock’s method. In this algorithm, the objective is sampled at a broader set of points than in the 
simplex-based methods as a way to avoid premature termination. There is also an element of random search 
involved. The ACSIM algorithm of Dixon [10] is a sophisticated direct search algorithm, combining ideas from 
the Nelder-Mead simplex method and the Complex method with elements of hem-stitching and quadratic 
modeling to accelerate convergence. 

In the special case of bound constraints, Spendley also suggested the expedient of simply setting to the 
corresponding bound any variable that was tending to go infeasible [29]. In [17], Keefer proposed a hybrid, 
feasible iterates algorithm for bound constrained minimization that uses the algorithm of Nelder-Mead in 
the interior of the feasible region and the method of Hooke and Jeeves at the boundary, since the pattern in 
the algorithm of Hooke and Jeeves conforms in a natural way to the boundary of the feasible region. In the 
case of linear constraints there is the algorithm of May [22], which is an extension of Mifflin’s derivative- free 
unconstrained minimization method in [23]. This algorithm also takes into account the particular geometry 
of the feasible region. 

Others have proposed modifications of the method of Hooke and Jeeves along the lines of feasible 
directions algorithms. These methods involve a limited calculation of sensitivity information to compute 
feasible directions at the boundary of the feasible region if the algorithm appears to have stalled. Klingman 
and Himmelblau [18] give an algorithm with a simple construction of a suitable feasible direction. The 
method of Glass and Cooper [12] is more sophisticated, and computes a new search direction by solving a 
linear programming problem involving a linear approximation of the objective and constraints, just as one 
would in a derivative- based feasible directions algorithm. 

Finally, we note the flexible tolerance method of Paviani and Himmelblau [14, 25]. This algorithm, based 
on the method of Nelder and Mead, alternatively attempts to reduce the objective and constraint violation, 
depending on the extent to which the iterates are infeasible. 

These proposals for direct search algorithms for constrained minimization, while they have often proven 
effective, have not been accompanied by any convergence analysis. A notable exception is May’s algorithm 
for linearly constrained minimization [22]; his sufficient decrease criterion for accepting steps enables him to 
prove global convergence. More recently, provably convergent, feasible iterate pattern search algorithms for 
bound constrained and linearly constrained minimization were developed in [19, 21]; we apply the analysis 
for bound constrained pattern search methods in the present work. 

2. The augmented Lagrangian method of Conn, Gould, and Toint. We begin by reviewing the 
augmented Lagrangian approach in [6]. To facilitate comparison of the pattern search approach with the 
original algorithm, we will adhere to the notation of [6] throughout. 

The augmented Lagrangian is 

(2.1) $(x; A, S, n) = f(x) + XiCi(x) + — V suCi(x) 2 . 

i— 1 ^ i— 1 

The vector A = (Ai, * ■ ■ , A m ) T is the Lagrange multiplier estimate, fi is the penalty parameter, and the entries 
sa of the diagonal matrix S are positive weights. The equality constraints of (1.1) are incorporated in the 
augmented Lagrangian $ while the simple bounds are left explicit. For a particular choice of multiplier X^ k \ 
penalty parameter fi^ k \ and scaling S^ k \ we define 
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Given an iterate , we define 

- V x $(x (fc) ;A ^,5^,//^). 

Conn, Gould, and Toint define the first-order Lagrange multiplier update to be 
( 2 . 2 ) A(x, A, 5, fi) = A + Sc(x ) //j. 

This is a form of the Hestenes- Powell multiplier update for the augmented Lagrangian ( 2 . 1 ). For the purposes 
of a pattern search augmented Lagrangian approach, which assumes no explicit knowledge of derivative 
information, one appears to have no choice other than some variant of the Hestenes- Powell multiplier update. 
All other multiplier update formulae (such as those discussed in [ 1 , 32]) require information about derivatives. 

The projection onto the convex set B = { x | ^ < u } will be denoted by P; it is defined 

component- wise by 

{ ti if Xi < £i 
Ui if Xi > Ui 

Xi otherwise. 

Given x E B and a vector v, we define 

P(x, v) = x — P[x — v\. 

Unless otherwise noted, we use || * || to denote the Euclidean vector norm or its induced matrix norm. 

We base our augmented Lagrangian pattern search method on Algorithm 1 of [ 6 ]. The original algorithm 
follows. 

Step 0 [Initialization]. An initial vector of Lagrange multiplier estimates A^°) is given. The positive 
constants rjo, < l, 7 i < 1 ,U 7 * <C 1 , 77 + 1 > , /3 U , a v , and are specified. The diagonal matrices 

Si and £ 2 , for which 0 < 5f 1 < 52 < 00 , are given (the inequalities are to be understood element-wise for 
the diagonal elements). Set = /i 0 , = min(//°\ 71 ), aA°) = u>o(cx^) a,A/ , = 7 ?o(<^°' , ) aT L and k = 0 . 

Step 1 [Inner iteration]. Define a scaling matrix for which S'f 1 < < 52 . Find x ^ E B 

such that 

(2.3) || P(x (/c) ,V x $ (fc) ) \\<J k \ 

If 

II C(l' fc >) \\<T) {k \ 

execute Step 2. Otherwise, execute Step 3. 

Step 2 [Test for convergence and update Lagrange multiplier estimates]. If || P(x^ k \ V x 3>^) || < 
u>* and || c(x^) || < 77 *, stop. Otherwise, set 

A( fc + 1 )=A(x( fc \A W.SW^W) 

= 

a (fc+l) _ m i n (^(^ + 1 ) ) -y 1 ) 

^(fc+1) = w {k) + 


increment k by one and go to Step 1. 



Step 3 [Reduce the penalty parameter]. Set 

A (fc+i) _ X (k) 

^ k+ 1) = T ■/!<*> 

Q,(fc+i) — min(/i^ fe+1 \7 1 ) 
u/ fc+1 ) =w 0 (a( fc+1) ) a “ 

V^ k+1) =m(a (k+1) ) a \ 

increment k by one and go to Step 1. 

3. Bound constrained pattern search algorithms. We next review the relevant features of the 
general pattern search method for the bound constrained problem 

, x minimize fix) 

(3.1) * 

subject to l < x < u. 

As noted in [19], a number of “classical” pattern search algorithms are suitable for bound constrained 
minimization, including 

• coordinate search with fixed step lengths [26], 

• evolutionary operation using composite designs ([2] and [4, 31]), 

• the original pattern search method of Hooke and Jeeves [15], and 

• the multidirectional search algorithm ([33, 34] and [9]) 

For a further discussion, see [19, 21]. 

3.1. The pattern. The index j will denote the iteration in a pattern search method. A pattern ptf) 
is a matrix Ptf) E Z nXPj , where p 3 > n + 1. There is no upper bound on pj. We partition the pattern into 
components 

pU) = [ rw Ltf) ]. 

We require that Ttf) E Z nxr J belong to a finite set of matrices T and that £tf) E Z nx ^ Pj ~ T ^ contains at 
least one column, a column of zeroes. The inclusion of a column of zeroes is simply a formalism to allow for 
a zero step, i.e., a;tf +1 ) = 

The matrices Ttf) must satisfy certain conditions, discussed more fully in [19, 21], that ensure that near 
the boundary of the bound constrained feasible region we always have a set of generators for any possible 
tangent cone. This, in turn, means that we can capture any feasible improvement in the objective. 

For the purposes of this discussion, the reader may assume that rtf) = [I — /], or, more generally, 

(3.2) r (i ) = [D® -£>tf)], 
where 

(3.3) = diag(d' j) ), i = 1, • • ■ ,n. 

This was the prescription for the pattern given in [19]. In [21] this condition is relaxed so that a bound con- 
strained pattern search algorithm can behave like a pattern search algorithm for unconstrained minimization 
in the interior of the feasible region or in a subspace of unbounded variables. 

At iteration j , given Atf) E 1R with Atf) > 0, we define a trial step to be a vector of the form 
gtf) = Atf)ctf^ for some i E {1, * • ■ ,Pj}, where denotes the ith column of Ptf) (i.e., Ptf) = [c^ * ■ • Cp}\)- 
We call a trial step feasible for (3.1) if (xtf) + s^) eB = {i | £ < x < u }. At iteration j, a trial 
point is any point of the form x^ = x^ + s^\ where xtf) is the current iterate. 
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3.2. The bound constrained exploratory moves. Pattern search methods proceed by conducting 
a scries of exploratory moves about the current iterate to choose a new iterate = x^ -f for 

some feasible step determined during the course of the exploratory moves. The following hypotheses 
on the result of the bound constrained exploratory moves allow a broad choice of exploratory moves while 
ensuring the properties required to prove convergence. By abuse of notation, if A is a matrix, y e A means 
that the vector y is a column of A. 

1. *0) g A (Apti) = [r^ LW)]. 

2. (x^ + s^) G £ = { x | £ < x < u }. 

3. If min { f(x^ + y) | y e A^T^ j \ x ^ + y € B } < /(x^), 
then f(x^ -f s^) < f(x^). 


Fig. 3.1. Hypotheses on the result of the bound constrained exploratory moves. 


3.3. The bound constrained pattern search method. Fig. 3.2 states the generalized pattern search 
method for minimization with bound constraints. To define a particular pattern search method, we must 
specify the pattern P^\ the bound constrained exploratory moves to be used to produce a feasible step s^\ 
and the algorithms for updating P^) and A 1 ^. 

Let x^ € B and A^ >0 be given. 

For j =0,1,"-, 

a) Compute f(x^). 

b) Determine a step using a bound constrained exploratory moves algorithm. 

c) If f(x^ + then x^ 1 ) = x^ + Otherwise xk’+U = x^\ 

d) Update and A^L 

Fig. 3.2. The Generalized Pattern Search Method for Bound Constrained Problems. 


3.4. The updates. The aim of the update of A^ is to force a strict reduction in /. An iteration with 
-j- s^) < f{x is successful ; otherwise, the iteration is unsuccessful. Note that to accept a step we 
only require simple , as opposed to sufficient , decrease. We cannot increase or decrease A^ in an arbitrary 
manner (as is detailed more fully in [19, 21]), but for the purposes of analyzing the augmented Lagrangian 
pattern search algorithm, the update of A^ can be summarized as 

(3.4) If /(x^ + $W) < f(x^) then A^ +1 ) > A^. 

(3.5) If f(x& + *0)) > /(x«>) then A^ +1 ) < AW. 

If an iteration is successful it may be possible to increase the step length parameter A^, but A W) is 
not allowed to decrease. Whereas if an iteration is unsuccessful, the step length parameter A^ must be 
decreased. Again, we refer the reader to [19, 21] for the details. 

4. The pattern search augmented Lagrangian method. At iteration k of the original augmented 
Lagrangian algorithm described in §2, we approximately solve the subproblem 

minimize $^(x) 

(4.1) 

subject to £ < x < u. 
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The degree to which this subproblem must be solved is given by (2.3). We adapt Algorithm 1 in [6] to 
pattern search by solving the bound constrained subproblem using a bound constrained pattern search 
method. However, pattern search methods do not have recourse to derivatives or explicit approximations 
thereof. For that reason we must replace the stopping criterion (2.3) with one that is appropriate to a pattern 
search method. 

We replace (2.3) with a new criterion on the size of the pattern. As we discuss in §5, we retain the con- 
vergence properties of the original Conn, Gould and Toint algorithm because the size of the pattern and the 
stationarity condition (2.3) are correlated, even though we do not have explicit control of || P(x ^ , V x 3>^) ||. 

We now state the augmented Lagrangian pattern search algorithm. At iteration k in the outermost loop 
of the algorithm, we will denote by the sequence of iterates produced in the solution of (4.1) via 

a bound constrained pattern search algorithm. We also assume that there exists d* such that \d[ k '^ | < d* 
for all k , where the are the diagonal entries in (3.3). This uniformity in the pattern search algorithms 
used in the successive minimization of the augmented Lagrangian is not at all restrictive. An obvious way in 
which to accomplish this is simply to choose for all k the same set T in the definition of the pattern search 
algorithms (see §3.1). 

In order to relate the stopping criterion in the pattern search solution of the subproblems to the multiplier 
estimates and the penalty parameter, we introduce the function 

= (i + II -Ml + Vm) 1 ’ 

We note that any function #(A, fi) such that 

^(A,^) = 0((I| A (I + l/^)' 1 ) 

as (|| A || 4- 1/fi) — > o° will suffice for the purposes of proving convergence. 

Step 0 [Initialization]. An initial vector of Lagrange multiplier estimates is given. The positive 
constants rjo , , wq , r < l,7i < 1,(5* <C 1,7?* 1,^,^,^, and j3 v are specified. The diagonal matrices 

S\ and 52, for which 0 < S^ 1 < S 2 < 00, are given (the inequalities are to be understood element-wise 
for the diagonal elements). Set /i^ = fi 0 , 71), = cJo(a^) a ‘*'> <5^ = 0(X^°\ /z(°))u;( 0 \ 

7^C°) = 770(0^ ) Qr? , and k = 0. 

Step 1 [Inner iteration]. Define a scaling matrix 5 ^ for which S^ 1 < 5^ < 52- Apply the bound 
constrained pattern search method to 

minimize 

(4.2) V ' 

subject to £ < x < u 

to find x W = x.( k '^ € B such that the pattern is sufficiently small, 

(4.3) A ( *^ < 

and we do not find an acceptable step in the part of the pattern p( k 'fi corresponding to r^’^, 

(4.4) /(*<**) + s&'rt) > f{x^) for all a G A^r^'>, 

The latter is the case, for instance, in the event of an unsuccessful step. 

If 

|| c(x< fe >)||<»7 (fc) , 

execute Step 2. Otherwise, execute Step 3. 
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Step 2 [Test for convergence and update Lagrange multiplier estimates]. If < 6 * and 

|| c(x^) || < T}», stop. Otherwise, set 

X {k+i) 

W (fe+D 

increment k by one and go to Step 1. 

Step 3 [Reduce the penalty parameter]. Set 

X (k+l) _ X (k) 

^k+l) = Ttl (k) 

Q,(*:+l) _ jaxa(fl^ k+1 \ 7j) 

w (*+ 1 ) = wo ( a ( fc + 1 )) Q - 
i = 0(\( k+1 \n {k+1) ) u {k+1) 

^( fe +D = 7 7o (a (A:+1) )^, 

increment k by one and go to Step 1. 

Note that we have replaced the stopping criterion (2.3) for the inner iteration of Algorithm 1 in [6] with 
(4.3)-(4.4), which are stopping criteria based on the size of the pattern, because we do not assume explicit 
information about the derivatives. The remaining modifications to Algorithm 1 in [6] are to correctly manage 
the sequence {£^}, which controls the stopping criteria we have introduced. The question now remains: 
having removed an exact specification of how inexact the solution of the subproblem can be (i.e., (2.3)), are 
the weaker conditions we have introduced (i.e., (4.3)- (4.4)) sufficient to guarantee that (2.3) will be satisfied 
asymptotically? An answer in the affirmative is provided in the next section. 

5. Convergence analysis. We now discuss the convergence properties of the augmented Lagrangian 
pattern search algorithm. As we shall see, altering the original algorithm by solving the bound constrained 
subproblem via pattern search does leaves the convergence properties of the original algorithm almost entirely 
unchanged. 

In [6], Conn, Gould, and Toint call a component of x ^ floating if 

£i < x\ k) - {V x <f>W)i < u { . 

For a convergent subsequence }, k £ K, with limit point x* they define the index set 

I\ ~ { i | x-^ are floating for all k € K sufficiently large and £ t < x’ < u, }, 

and let /l(x) denote the corresponding columns of the Jacobian of c(x), where A(x) is the entire Jacobian 
of c(x). 

The following assumptions are made in [6]. 

AS1. The functions f(x) and c(x) are twice continuously differentiable for all x € B. 


= A(x<*>,A W,S (fc \// fc) ) 
= niin(/i (fe+ 1 ) ,7 1 ) 

= wW(a (fc+1 >)^ 

= <9(A ( fc+1 ),p( fe + 1 )) 

= r } {k) (a {k+1) f r >, 



AS2. The iterates {x^} considered lie within a closed, bounded domain 

AS3. The matrix A(x*) has column rank no smaller than m at any limit point x* of the sequences 
{x^ } considered in this paper . 

In addition, in order to be assured that a bound constrained pattern search algorithm applied to the 
subproblem (4.2) will find an iterate satisfying (4.3) (4.4), we assume the following. 

PS1. For a given k, the set B D { x | $^)(x) < <&( fc )(x^’ 0 )) } is compact. 

That is, we assume compactness of the set of x E B for which the augmented Lagrangian is less than the 
value of the augmented Lagrangian at the point at which we begin the solution of the subproblem. This 
is not a particularly restrictive assumption, as we discuss further in the context of inequality constrained 
minimization in §6, but it is necessary to ensure convergence of any pattern search method applied to the 
subproblem (4.2). 

Under hypothesis (PS1), we are assured that in the inner iteration (the pattern search minimization of 
the bound constrained augmented Lagrangian), 

liminf =0 

j—>+oo 

(see [19, 21]), so the termination criterion (4.3) eventually will be satisfied. Moreover, the update rules 
(3.4) (3.5) only allow to be decreased at unsuccessful steps, where (4.4) holds. Thus both termination 

criteria (4.3) and (4.4) will eventually be satisfied, the pattern search solution of the augmented Lagrangian 
subproblem will halt, and the overall iteration of the pattern search augmented Lagrangian algorithm is 
well-defined. 

5.1. The relationship between the pattern size and stationarity. For convenience, let 

q (k,j) = 

The following result is the key to analyzing the augmented Lagrangian pattern search method. 
Proposition 5.1. There exists C 5 .i, independent of k, such that 

|| P{x {k \V x & k) ) || < Cs.i 


for all k. 

Proof Given x^ = for some j. By design we have d* > 0 such that < d* for all i , j, 

and k , where d\ k '^ is as defined in (3.2). 

First suppose 

(5.1) A<« > » K 

Then (5.1), (4.3), and the rule for updating 6^ in either Step 2 or Step 3 give us 

|| <7 (M Hoc < d*A (k ' j) < d*5 (fc) < d*a/ fe) 


and so 
(5.2) 

On the other hand, suppose 


|| q( k ' j ) || <n*<Tu; (fe) . 




II q (kJ) 


d* 
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The proof of Proposition 5.2 in [19] shows that if < || ||oc/d*, then there is a trial step E 

suc h that E B and 

(5.3) V«* w (x (fcj ’ ) ) T ^ fc,i) < -n-i || q ikJ) |||| s\ k>j) ||. 

The stopping criterion (4.4) means that 

(5.4) 0 < $ {k \x (k) + sf J) ) - $ (fc) (x (fc) ). 

At the same time we have 

(5.5) $ w (x (fc) + s\ k,j) ) - $W(x (fe) ) = V x $ (fc) (O r sf’' 7) 

for some £ in the line segment (*^,^ + .s^’^ ) connecting x^ and + s\ k '^ . Thus from (5.4), (5.5), 

and (5.3) we obtain 

0 <*W(*« +,{**«) _®«(*W) 

= Vx$ (fc) (x (fe) ) r s t (fc,i) + (V x $ (fc) (£) - V x $ (fc) (x (k) )) T sf ' j) 

< -n"*| q {k ' j) llll sf J) II + II V x c&< fc >(£) - V x & k \x™) |||| s\ kJ) ||, 

which yields 

(5.6) || ql k '» || < n* || V x $ (fc) (£) - V x * (fe) (x (fc) ) II- 
Applying the mean- value theorem again, for some £ E (x^,£) we have 

V«* w (0 - V x $ (fc >(x^) = V^* W (C)(€ - * (fc) ), 
so 

II V x *«(fl - V.*W(afW) II < II VL^ (fc) (0 llll e - II 
(5-7) < II V^^ (fc) (C) llll »i kJ) II- 

Now, 

m _ m 

VL^ (fc) (0 = VL/(C) + £A< fc) V 2 c(0 + -TjK , (Vc(C)5Vc(C) T + E*« c *(OV a c»(C)). 

1=1 M 1=1 

By construction, — > 0, so — ► 0, so by (AS2), £ lies in a compact subset that is independent of k . 

Furthermore, the bound < S 2 is independent of k. Thus we can find M, independent of fc, such that 

II VL$ (fc) (0 II < M + M|| A« || + Jlf-h 

Returning to (5.7) we have 

(5.8) || V x 4>«(e) - V x & k \xW) || < M (l + || || + || •j* J) || . 

Thus from (5.6), (5.8), the fact that and (4.3) we have 

|| || < n* || V x $ (fc) (£) - V x $ (fc) (x (fc) ) || 

<n*Jlf(l + || || + ^-) II 4^ II 

< n k d*M fl + || A (fc) || + A (fe,j) 

< n*d*M (l + || A<*> || + -JA 
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Finally, the rule for updating S'' k) in either Step 2 or Step 3 gives us 
(5.9) || || < n*d’Moj {k) . 

Combining (5.2) and (5.9) yields the proposition. □ 

5.2. Convergence results. Proposition 5.1 means that the asymptotic behavior of || P(x^ k \ V x $^) || 
in the augmented Lagrangian pattern search algorithm is like that of the same quantity in the original algo- 
rithm. This, in turn, allows us to piggy-back the convergence analysis for the augmented Lagrangian pattern 
search algorithm on that for the original augmented Lagrangian algorithm in [6]. Because of Proposition 5.1 
the original proofs of all these results still hold. 

The first convergence result corresponds to Theorem 4.4 and Lemma 4.3 in [6]. Let 

771 

9i(x;A) = V/(x) + ^A i Vc i (x), 

i= 1 

which is the gradient of the Lagrangian with respect to the constraints c* (x) only. 

Theorem 5.2. Assume that (ASl) holds. Let x* be any limit point of the sequence {x^} generated by 
the augmented Lagrangian pattern search algorithm for which (AS2) and (ASS) hold and let K be the set of 
indices of an infinite subsequence of the x^ whose limit is x* . Then 

(i) c(x*) = 0. 

(ii) x* is a Karush- Kuhn- Tucker point ( first-order stationary point) for the problem (1-1), X* is the 
corresponding vector of Lagrange multipliers, and the sequence {X(x^ k \ X^ k \ S^ k \ p^)} converges to X* for 
keK . 

(Hi) There are positive constants a < 22 , s \ and an integer ko such that 

|| A(x (fc) ,A (fc \S (fc \M (fc) )- A* || < ai u>W +a2 y X W _ x * || 


and 

|| c(x (fc) ) || < + M «|| A (fc) - A* 1 + a 2/ /*> || x W - x* ||) 

for all k > ko, (keK). 

(iv) The gradients converge to <?l(£*; A*) for keK. 

As in [6], under additional assumptions we obtain stronger results. Following [6], if J\ and J 2 are any 
index sets, and Hl(x*, A*) is the Hessian of the Lagrangian, then Hl(x*, X*)[j lf j 2 ] is the matrix formed by 
taking the rows and columns of Hl(x*,X*) indexed by J\ and J 2 , respectively, while A(x*)[j 1 ] is the matrix 
formed by taking the columns of A(x*) indexed by J\. We then make the following assumptions. 

AS4. The second derivatives of the functions f(x) and the c»(x) are Lipschitz continuous at all points 
within £L 

AS5. Suppose that (x*, A*) is a K arush- Kuhn- Thicker point for the problem (LI) and that 

J\ — {i | (9l(x*; A*))i = 0 and U < x\ < U* } 

J 2 = { i | (9l(x*\ A*))i = 0 and (x* — ii or x * = Ui) }. 

Then we assume that the matrix 

A*)[J,J] (j4(x*)[.7]) T 

^(**)[J1 0 
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is nonsingular for all sets J, where J is any set made up from the union of J\ and any subset of J 2 . 

The next result from [6], which also holds for the augmented Lagrangian pattern search algorithm, is 
Lemma 5.1. This result relates the convergence of the iterates to the error in the multipliers, a relationship 
characteristic of augmented Lagrangian methods [1, 32]. Again, the proof in [6] holds for the pattern search 
variant because of Proposition 5.1. 

Lemma 5.3. Suppose that (ASl) holds. Let {x^} c B, k E K, be a subsequence which converges to 
the Karush- Kuhn- Tucker point x* for which (AS 2 ), (AS 4 ), and (ASS) hold , and let X* be the corresponding 
vector of Lagrange multipliers. Assume that {A^}, k e K, is any sequence of vectors, that {S^}, k € K, 
is any sequence of diagonal matrices satisfying 0 < 5f 1 < S^ < S 2 < oo, and that {//*)}, k e K, form 
a nonincreasing sequence of positive scalars, so that the product p^\\ X ^ — X* || converges to zero as k 
increases. Now, suppose further that 

|| P(* (fc \V** (fc) ) || <u; ( *\ 

where the are positive scalar parameters which converge to zero as k G K increases. Then there are 
positive constants Ji, a$, a 4, 05, a 6 , and s 1 and an integer value ko so that if < Ji then 

(5.10) || X W — x * || < a 3W ( fc ) +a 4 /i (fe) || A<*> - A* || 

|| A(xW,A — A* || < a 5 w w +a 6 /i W || A (*) _ \* || 

and 

(5.11) || c(x (fc) ) || < Sl (a 5 u( k) ti {k) + (/i {fc) + a 6 ( M <*>) 2 )|| A< fc > _ \* ||) 

for all k > ko } (k e K). 

The following is Corollary 5.2 in [6]. 

COROLLARY 5.4. Suppose that the conditions of Lemma 5.3 hold and that A^ +1 ) is any Lagrange 
multiplier estimate for which 

|| A ( * +1 > - A* || < dull *« - x* || + a 17 u/ fc > 

for some positive constants ai6 and a\y and all k 6 K sufficiently large. Then there are positive constants 
Ji, as, 04 , as, a§, s\ and an integer value k 0 so that if < Ji then (5.10), 

|| x^+1) _ a* || < o 5 u/ (fe) + a 6 M (fc) || A (it > - A* ||, 
and (5.11) hold for all k > ko, (k € K). 

We also inherit the following result indicating that we may generally expect the penalty parameter to 
remain bounded away from zero. This is Theorem 5.3 in [6]. Taken together with the convergence of the 
multiplier estimates, this means that the stopping tolerance for the inexact minimization of the augmented 
Lagrangian is decreasing at the same rate as in the original algorithm. However, in §6 of [6] the authors 
show that in the case of non- unique limit points one can have — ► 0, in which case the stopping tolerance 

5 k decreases more like (m^) 2 - 

Theorem 5.5. Suppose that the iterates {x^} of the augmented Lagrangian pattern search algorithm 
converge to the single limit point x* , that (ASl), (AS2), (AS 4 ), and (ASS) hold, and that a v and (3^ satisfy 
a v < min(l,a w ) and < min(l, f3J). Then there is a constant \i > 0 such that > fi for all k. 
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The proof of Theorem 5.5 makes use of the fact that || P(x^ k \ V x 3>^) || = 0(u; fc ), whereas the proofs 
of the preceding convergence results require only that 

|| || ->o. 


Finally, we have the following result on the rate of convergence of the outer iteration, corresponding to 
Theorem 5.5 in [6]. 

Theorem 5.6. Under the assumptions of Theorem 5.5, the iterates x ^ and the Lagrange multiplier 
estimates A^ of the augmented Lagrangian pattern search algorithm are at least R-linearly convergent with 
R-factor at most where ft = min [71 , p] and where p is the smallest value of the penalty parameter 

generated by the algorithm in question. 

6. Application to inequality constrained minimization. Special consideration is due to the gen- 
eral problem 


minimize f{y) 

(6.1) subject to g(y) < 0 

£<y <u, 

converted into the form (1.1) via the introduction of non-negative slack variables: 


( 6 . 2 ) 


minimize f(y) 

subject to g(y) + z = 0 
£ < y < u 
z > 0. 


The augmented Lagrangian associated with (6.2) is 

1 m 

(6.3) ®{y, z; A, S, ft) = f(y) + A T {g(y) + z) + — ^ sa(gi(y ) + Zi) 2 . 

^ i=\ 

Explicit equality constraints may also be present in (6.1); we ignore them here for brevity. 

The introduction of slacks increases the dimension of the bound constrained subproblem that we must 
solve at each outer iteration. Such increases in dimension usually cause a degradation in performance for 
pattern search methods. However, we can avoid this increase in dimension because of the simple way in 
which the slacks z enter into (6.3). A standard approach [1, 27] is to note that given y , we can minimize 
4>(y, z; A, S, /z) explicitly in z for z > 0. Doing so leads to a subproblem problem in y alone; 


(6.4) 


minimize 4>(y, z(y)\ A, S', p) 
subject to £ < y < u, 


where 

m 

®{V, z(y)\ A, S, ft) = f(y) + — (max(0, Aj + —gi(y )) 2 - A, 2 ). 

Z Sa (l 

The multiplier update formula (2.2) is also modified: 


\i(x, A, 5, p) = max(0, A^ + SuCi(x)fp) y i = 1, ■ * ■ , m. 


See [1] for further discussion. The reduced augmented Lagrangian $(y, z{y)\ A, S, p) has Lipschitz first 
derivatives. Moreover, if the feasible region for the original problem (6.1) is compact (e.g., if there arc upper 
and lower bounds on all the components of y), then the feasible region for (6.4) is also compact, so we may 
be assured of convergence of a bound constrained pattern search algorithm applied to (6.4). 
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